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INFORMATION PROVIDING METHOD AND APPARATUS, AND 



INFORMATION RECEPTION APPARATUS 



BACKGROUND OF THE INVENTION 



The present invention relates to a multimedia 
information providing method and apparatus for 
providing video, music, and text data to many and 
unspecified users through the Internet, etc., and a 
multimedia information reception apparatus for 
receiving the video, music, and text data. More 
particularly, the present invention relates to a 
multimedia information providing method and apparatus 
and a multimedia information reception apparatus for 
selecting only information of user's interest from a 
number of multimedia information and providing 
individual information to the user. 

This application is based on Japanese Patent 
Application No. 10-372746, filed December 28, 1998, the 
entire content of which is incorporated herein by 
reference . 

In recent years, growth of information 
infrastructures is boosting opportunities for 
distributing home many multimedia information through 
CATV (cable television broadcasting), digital satellite 
broadcasting, or the Internet. A variety of programs 
are provided, and the number of service channels has 
reached an order of several hundred or several thousand. 
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Therefore, it is becoming difficult for a user to 
appropriately select information from the several 
hundred or several thousand channels or tens of 
thousands or more programs in the channels. 

To solve this problem, a receiver device for 
automatically recording programs of user's interest 
using the information of an electronic program list 
sent from a broadcasting station has been proposed 
(e.g., "video device" disclosed in Jpn. Pat. Appln. 
KOKAI Publication No. 7-135621). 

This proposed device selects programs that may be 
of interest for a user from the information of an 
electronic program list on the basis of keywords 
registered in advance and automatically filters 
programs in units of users. 

To prepare an attractive program providing program, 
a program provider wants to know the types of programs 
viewed by viewers. Conventionally, a provider raises 
monitors, lets them fill out a questionnaire, and 
gathers the results to know programs watched by the 
monitors. However, with the questionnaire of fill-out 
type, only rough information representing whether or 
not a viewer has watched a certain program can be 
obtained. 

In a conventional system for automatically 
selecting a program from an enormous number of programs 
provided by a program provider in accordance with a 
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personal taste, selection is just roughly done in units 
of programs. Consider a program such as a news show or 
a variety show. In such programs, one program is 
constructed by units of "topics" or "corners". Quite 
often, user's interest is only in some of images in one 
program. However, in automatic recording in units of 
programs, one program is entirely selected and recorded 
from the beginning to the last. The user cannot know 
the position of information of his/her actual interest 
unless he/she watches the entire program. Hence, even 
when a program is selected and recorded by filtering, 
the user must watch the recorded program from the 
beginning to the last, wasting the recording medium and 
user's time. 

Filtering may omit CMs contained in a program. 
When a broadcasted program is not a pay TV program but 
a free program for which the ad rate is the source of 
revenue, whether the viewers actually watch CMs or not 
is an important factor for the program provider in 
soliciting advertisement. Hence, to exclude CMs from 
the program content poses a serious problem. 

In addition, conventional audience rating survey 
is done in units of programs and is therefore 
insufficient to precisely grasp the users' tastes and 
the like. 

BRIEF SUMMARY OF THE INVENTION 
Accordingly, it is an object of the present 




invention to provide the following information 
providing method and apparatus, information reception 
apparatus, and data structure. 

It is the first object of the present invention to 
provide an information providing method and apparatus 
and an information reception apparatus capable of 
appropriately selecting and providing portions of 
user's actual interest from a number of multimedia 
information instead of filtering in units of programs. 

It is the second object of the present invention 
to provide an information providing method and 
apparatus and an information reception apparatus 
capable of appropriately selecting and providing 
portions of user's actual interest from a number of 
multimedia information instead of filtering in units of 
programs, in which a commercial message that the 
program provider wants a viewer to watch is surely 
provided . 

It is the third object of the present invention to 
provide an information providing method and apparatus 
and an information reception apparatus capable of 
appropriately selecting and providing portions of 
user's actual interest from a number of multimedia 
information instead of filtering in units of programs, 
in which user's viewing history is recorded, and a user 
profile representing user's taste can be updated in 
accordance with the viewing history. 



It is the fourth object of the present invention 
to provide a data structure of describing the above 
user profile used in the information providing 
apparatus and the information reception apparatus. 

According to the present invention , there is 
provided an information providing method comprising: 

adding program feature data to multimedia 
information in units of parts of the multimedia 
information to form a program database; 

searching for partial information which accords 
with user profile data from the multimedia information 
based on matching between the user profile data and the 
program feature data; and 

providing the searched partial information to a 

user . 

According to the present invention, there is 
provided an information providing apparatus comprising: 

a first database configured to store multimedia 
information; 

an analyze section configured to analyze the 
multimedia information stored in the first database 
using at least one analysis method of moving image 
analysis, acoustic/speech analysis, and text analysis; 

a second database configured to store program 
feature data which is obtained in units of parts of the 
multimedia information or externally inputted; and 

a search engine configured to search for program 



feature data from the second database in accordance 
with user profile data, and select partial information 
from the multimedia information stored in the first 
database in accordance with searched program feature 
data . 

According to the present invention, there is 
provided an information reception apparatus connected 
to an information providing server having a database 
which stores multimedia information and program feature 
data which is an analysis result of at least one of 
moving image analysis, acoustic/speech analysis, and 
text analysis or externally inputted, comprising: 

a search engine configured to search for 
predetermined program feature data from the database 
and select partial information from the multimedia 
information stored in the database in accordance with 
searched program feature data. 

According to the present invention, there is 
provided an information describing method comprising: 

classifying information items into plural groups 
of information items relating to personal information 
of a user, some of the groups of information items 
including plural subgroups; and 

describing each information items in the group or 
the subgroup in an order according to a priority of the 
information item which is determined for each user. 

According to the present invention, corresponding 



partial information can be selected on the basis of a 
user profile data. 

It is possible to select such a commercial message 
in accordance with the user profile data a commercial 
message that a program provider wants a viewer to watch 
if the commercial message is also stored in the 
database as in the same manner as the multimedia 
information . 

Additional objects and advantages of the present 
invention will be set forth in the description which 
follows, and in part will be obvious from the 
description, or may be learned by practice of the 
present invention . 

The objects and advantages of the present 
invention may be realized and obtained by means of the 
instrumentalities and combinations particularly pointed 
out hereinafter. 

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 

The accompanying drawings, which are incorporated 
in and constitute a part of the specification, 
illustrate presently preferred embodiments of the 
present invention and, together with the general 
description given above and the detailed description of 
the preferred embodiments given below, serve to explain 
the principles of the present invention in which: 

FIG. 1 is a block diagram showing the basic 
arrangement of an information providing apparatus 



according to the first embodiment of the present 
invention; 

FIGS. 2A to 2E are views showing the data 
structure of a user profile; 

FIG, 3 is a flow chart showing the operation of a 
feature extraction section; 

FIG* 4 is a view showing an example of an 
extracted program feature; 

FIG. 5 is a flow chart showing the operation of a 
search engine; 

FIG. 6 is a view showing an example of a search 
result; 

FIG. 7 is a flow chart showing the operation of a 
link section; 

FIG. 8 is a view showing an example of a display 
window generated by the link section; 

FIG. 9 is a view showing an example of CM display; 

FIG. 10 is a view showing another example of CM 
display; 

FIG. 11 is a block diagram showing the first 
modification of the first embodiment applied to a 
server /client system; 

FIG. 12 is a block diagram showing the second 
modification of the first embodiment applied to a 
server/client system; 

FIG. 13 is a block diagram showing the third 
modification of the first embodiment applied to a 



server /client system; 

FIG, 14 is a block diagram showing the basic 
arrangement of an information providing apparatus 
according to the second embodiment of the present 
invention; 

FIG • 15 is a block diagram showing the first 
modification of the second embodiment applied to a 
server /client system; 

FIG, 16 is a block diagram showing the second 
modification of the second embodiment applied to a 
server /client system; 

FIG. 17 is a block diagram showing the third 
modification of the second embodiment applied to a 
server /client system; 

FIG. 18 is a flow chart showing the operation of a 
viewing history recording section; 

FIGS. 19A and 19B are views showing an example of 
a viewing history; and 

FIG. 20 is a flow chart showing the user profile 
update operation. 

DETAILED DESCRIPTION OF THE INVENTION 

A preferred embodiment of an information providing 
apparatus according to the present invention will now 
be described with reference to the accompanying 
drawings . 
First Embodiment 

FIG. 1 is a block diagram showing the basic 



arrangement of an information providing apparatus 
according to the first embodiment of the present 
invention. This apparatus has a multimedia information 
database 101 , commercial message (CM) database 108, 
program feature database 103 , CM feature database 109 , 
and user profile database 104, which are constructed by 
media capable of random access. The databases are 
classified for the descriptive convenience. Physically, 
one database may be used. 

The multimedia information database 101 stores a 
number of multimedia information to be provided. The 
CM database 108 stores a number of CM information to be 
provided together with free programs. A CM feature 
representing the contents of CM information is stored 
in the CM feature database 10 9 for every CM information. 

The pieces of multimedia information are a number 
of programs provided by an information provider such as 
a broadcasting station or the Internet. Analog data is 
converted into digital data in advance and then stored 
in the multimedia information database 101 and managed. 
The digital data can be MPEG-2 compressed data or DV 
compressed data. The multimedia information have 
"title names" in units of programs and "frame numbers" 
in units of frames in each program and are stored in a 
medium, e.g., a hard disk which can be accessed from an 
arbitrary position in accordance with a given title 
name and frame number. The medium is not limited to 



the hard disk and may be another medium such as a 
DVD-RAM (ROM) capable of random access. The multimedia 
information need not maintain the image size and 
quality of the original analog data. A compression 
scheme such as MPEG-.l or MPEG-4 that saves the image 
capacity may be employed depending on the application 
intended. 

The output from the multimedia information 
database 101 is supplied to a feature extraction 
section 102. The feature extraction section 102 
performs predetermined analysis for all information 
held in the multimedia information database 101 , sorts 
the information in accordance with the analysis result , 
and adds program features representing the contents in 
units of sorts (e.g., in units of frames). The program 
features are managed by the program feature database 
103 in units of sorts. 

CM features (CM program units) are known in 
advance. When CM information is stored in the CM 
database 108, a corresponding CM feature is stored in 
the CM feature database 10 9. However, a program 
feature is obtained by storing new program information 
in the multimedia information database 101, reading out 
the information from the database 101, and then 
analyzing the information. The program feature may be 
separately obtained and input to the program feature 
database 103 by the operator using a keyboard 110. 



When both the automatic program feature analysis and 
the determination by the operator are used, a more 
appropriate feature can be added to the program 
information (addition of an index). 

The feature extraction section 102 performs video 
analysis and acoustic/speech analysis for multimedia 
information . 

For video analysis, a technique of determining the 
video data structure on the basis of information of a 
cut with an instantaneous change in a video scene or 
camera movement (pan or zoom) using moving image 
analysis that has conventionally been studied, and 
obtaining the feature of the video data can be used. 

The position where the scene instantaneously 
changes can be detected by comparing the similarity 
between frame images of the video data. The similarity 
can be obtained by calculating the histogram of the 
frequency of a color in each image and comparing the 
histograms. A portion with low similarity is a point 
where the scene instantaneously changes. 

To provide a camera movement parameter, optical 
flows representing the positions of movement of pixels 
are obtained from two images. Assuming that most 
optical flows are obtained from the background, the 
movement of the camera is calculated on the basis of 
dominant optical flows. 

When the camera is panning, most optical flows 



appear parallel to each other. When the camera is 
zooming, optical flows point in the direction of a 
certain point. Details are described in reference (1), 
Hirotada Ueno, Takafumi Miyabu, and Satoshi Yoshizawa, 
"Proposal of interactive Video Editing Scheme Using 
Recognition Technology" , IECE Papers (D-II), VOL. 
J75-D-II, No. 2, pp. 216 - 225 and reference (2), 
Masahiro Shibata, "Video Contents Description Model and 
Its Application to Video Structuring", IECE Papers 
(D-II), VOL. J78-D-II, No. 2, pp. 754 - 764. 

With acoustic /speech analysis, music and human 
voice can be separated from each other because music 
has few mute portions and frequency components that 
are absent in human voice, and voice data can be 
discriminated because human voice has characteristic 
features reverse to those of music, and male voice and 
female voice have a pitch difference. 

Details of the method of identifying male voice 
and female voice are described in reference (3), 
Keiichi Minami, Akihito Akutsu, Hiroshi Hamada, and 
Yoshinobu Sotomura, "Video Indexing Using Sound 
Information and Its Application", IECE Papers (D-II), 
VOL. J81-D-II, No. 3, pp. 529 - 537, and a detailed 
description thereof will be omitted. 

With this method, video data is sorted from the 
video information and speech information, and a feature 
can be added to each sort. 



For example, sound data is analyzed to separate a 
music portion from a portion of male/female voice. 
Then, a video scene associated with the sound data is 
discriminated into a scene associated with the music 
portion, a scene associated with male voice, and a 
scene associated with female voice, and features are 
determined for the respective scenes. 

If character data associated with video data 
accompanies the video data, the text is analyzed to 
determine the feature. In the U.S.A. , video data 
contains character data called a closed caption. If 
such data can be used, text analysis using the 
conventional natural language processing technology can 
be performed to determine the feature according to the 
contents . 

That is, on the basis of character data 
accompanying an image, a feature based on the analysis 
result of character data contents associated with video 
data can be added in units of sorts. 

The user profile database 104 is a file in which 
information (user profile) of the taste or the field of 
interest of each user is registered, and managed in 
units of users. The user profile is prepared by 
inquiring the user or obtaining information through a 
questionnaire in advance. As shown in FIGS. 2A to 2E, 
the user profile has text information and includes 
keywords representing the taste and the field of 
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interest of a user. FIG, 2A shows taste information 
associated with the type of programs, FIG. 2B shows 
taste information associated with the contents of 
programs, FIG. 2C shows taste information associated 
with production of programs, FIG. 2D shows the personal 
profile, and FIG. 2E shows keywords/key phrases 
representing the taste. FIG. 2A shows information of 
the program categories or genres such as suspense, 
drama, documentary, sports, variety, and news. FIG. 2B 
shows the types of scenes in one program. For example, 
a movie has information of favorite scenes such as an 
action scene, love scene, and climax scene. A news has 
information of politics, economy, sports, city news, 
and the like. FIG. 2C shows information of persons who 
produce programs, e.g., movie directors, actors, 
actresses, composers of music used in movies, lyric 
writers, and arrangers. Information of production 
areas are also included. FIG. 2D shows personal 
information such as the age, sex, occupation, 
birthplace (or home town), and birthday. FIG. 2E shows 
keywords and key phrases representing the taste of the 
user, e.g., various keywords and key phrases including 
a favorite food , favorite matter , favorite sport , hobby , 
and favorite proverb. In FIGS. 2A to 2E, a number in 
parentheses represents the number of items of taste. 
If a plurality of items are listed, the items are 
arranged in the order of priority. 



A search engine 105 searches the program feature 
database 103 and CM feature database 109 to select a 
feature matching the user profile in the user profile 
database 104, This makes it possible to find out the 
information portion of user's interest. To search for 
that portion, a matching feature is detected on the 
basis of the keywords in the user profile. In this 
keyword matching, features matching keywords similar to 
the user profile can also be detected using a thesaurus 
(dictionary of synonyms or taxonomy, or index for 
information search). The thesaurus also includes a 
dictionary considering differences in usages of 
language between countries or areas or gaps between 
sexes or generations (i.e., dictionary for eliminating 
the differences or gaps). 

With the search engine 105, associated video data 
can be finely specifically identified/searched in units 
of scenes, units associated with speech data, or units 
associated with character data, so a partial image of 
each user's interest can be selected and extracted. 

The search engine 105 supplies the search result 
to a link section 106. 

The link section 106 processes the information 
to reproduces it. That is, the link section 106 
associates the information in the CM database 108 and 
the result of search and reproduce the partial image 
according to the user profile. 



A display section 107 displays the image 
reconstructed by the link section 106. The display 
section 107 includes a loudspeaker for reproducing 
music information . 

An outline of the basic arrangement of this system 
has been described above. 

Methods of implementing the individual processing 
will be described below in detail. 

Details of processing by the search engine 105 
will be described with reference to FIG. 3. FIG. 3 
shows the flow of processing so as to explain details 
of processing by the feature extraction section 102. 

The feature extraction section 102 can analyze all 
multimedia information stored in the multimedia 
information database 101, analyze each information not 
in units of programs but in units of frames, and obtain 
a feature. 

Multimedia information contains not only image 
data but also sound and text data. Hence, analysis of 
multimedia information is performed in three steps: 
text analysis, moving image analysis, and acoustic/ 
speech analysis. The processing order is not 
particularly limited . 

For text analysis, closed caption information in 
the video data is extracted (steps SI and S2), 
morphemes are analyzed (step S3), and keywords are 
analyzed on the basis of the morpheme analysis result 
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(step S4). This analysis is performed for all video 
programs in the multimedia information database 101. 

For moving image analysis, a cut of a moving image 
in video data is detected (steps SI and S5), the camera 
movement parameter is extracted (step S6), and the 
video data is segmented on the basis of the camera 
movement parameter (step S7). This analysis is 
performed for all video programs in the multimedia 
information database 101. 

For acoustic/speech analysis, acoustic 
identification is performed in video data (steps SI and 
S8), speech recognition is performed (step S9), and 
keywords are extracted on the basis of the recognition 
result (step S10). This analysis is performed for 
all video programs in the multimedia information 
database 101. 

Text analysis, moving image analysis, and 
acoustic/speech analysis produce analysis results. 

By video analysis according to these procedures, 
various feature information are obtained in association 
with the multimedia information. The pieces of feature 
information are processed by high-level integration 
processing (step Sll) of integrating the individual 
information. 

For text analysis, moving image analysis, and 
acoustic/speech analysis, conventionally known analysis 
technologies can be used, as has already been described 



above . 

For example, in text analysis, a closed caption 
contained in video data is extracted, and the roles of 
words are analyzed by morpheme analysis. An important 
keyword such as a proper noun describing a scene is 
extracted from the words. As the keyword, not only a 
proper noun but also information representing a high 
frequency of occurrence is also used. 

In moving image analysis, video data is segmented 
by extracting a scene with an abrupt change or camera 
movement information (reference (1)). In acoustic/ 
speech analysis, music data and speech data are 
separated by speech identification, male voice and 
female voice are separated by speech recognition 
(reference (3)), and a keyword is extracted using 
speech recognition . 

Integration processing aims at storing feature 
information obtained by the individual processing as a 
database in association with each other and integrating 
the feature information to generate new feature 
information. 

For example, processing of associating individual 
processing is performed in the following way. 

Assume that processing is to be performed in units 
of segmented video data, and a keyword as an important 
proper noun is present in the video data. Even when 
the keyword is obtained from the caption (comment or 



explanation) , video frames corresponding to the 
position of the keyword cannot be accurately known. 

The position of the keyword is identified using 
speech recognition, and the keyword is added to a 
partial image at a position with consecutive speech 
data as a feature. 

The analysis result is generated as a table as 
shown in FIG. 4. In the table shown in FIG. 4, the 
title of the program is "news", keywords as features 
representing the characters or situation are "politics 
"economy", and "weather forecast", and "0:00 - 0:05", 
"0:15 - 0:16", and "0:23 - 0:25" are picked up as 
window appearance times (frames) associated with the 
respective keywords. That is, video data is segmented 
in reference to time (frames) in units of program 
titles, and important keywords (features) appearing in 
the frames are added to form a table. 

Details of processing by the search engine 105 
will be described next with reference to FIG. 5. 
Search associated with program information will be 
described below. This also applies to search 
associated with CM information. The search engine 105 
looks up information in the program feature database 
103 and user profile database 104 to extract features 
of user's interest, thereby selecting corresponding 
partial video data. 

Keywords are selected from the user profile 



database 104 one by one, and associated words are 
picked up using the thesaurus dictionary (steps S21 
and S22 ) • 

After picking up the associated words, the picked 
up associated words are compared with keywords 
represented in the features stored in the program 
feature database 103. If a word and keyword match with 
each other, information representing the position of 
the partial video data and the title to which the frame 
belongs is recorded (steps S23, S24 and S25). In 
keyword matching, if the same associated word recurs, 
it is compared upon each occurrence. 

Processing by the search engine 105 has been 
described above in detail. 

FIG. 6 shows an information example of partial 
video data acquired by keyword matching and registered 
in step S25. In this case, one keyword in the user 
profile database 104 is "shopping". Information of 
shopping are searched for using thesaurus data, and 
keywords such as "department store", "bakery", etc., 
are selected and compared with keywords in the program 
feature database 103 to provide information of 
corresponding video data as shown in FIG. 6. 

The above description has been made about only 
selection of multimedia information. CM information 
can be selected in the same way as described above. 

Details of the link section 106 will be described 



next with reference to the flow chart shown in FIG, 7. 
The link section 106 obtains information shown in 
FIG, 6 from the search engine 105 and constructs, from 
these information, a display window (index window) for 
providing information. In this case, thumbnail images 
as shown in FIG. 8 are displayed. 

First, it is determined whether processing for all 
keywords is ended (in the example shown in FIG. 8, 
keywords are "shopping", "public facility", 
"transportation/bank", and "health/hospital") . If 
processing is not ended, processing is continued (step 
S31). Partial video data selected in association with 
one keyword is acquired from the multimedia information 
database 101 (step S32). To acquire partial video data 
from the multimedia information database 101 by random 
access at a sufficiently high speed, time code 
information (frame information) can be directly used. 
Otherwise, a copy or partial video data, partial video 
data with a reduced window size, or a copy of partial 
video data using a different compression ratio or 
compression scheme is acquired. 

One or a plurality of frames of acquired partial 
video data are acquired as representative images (step 
S33) and used as materials to construct the window. 
The feature of each representative image is associated 
with the representative image, and the representative 
image is associated with the partial video data (steps 



S34 and S35). Information of the representative image 
is described using the HTML (step S36). 

When partial video data selected in accordance 
with a keyword is processed, the next keyword is 
processed. Otherwise, the above processing is repeated 
( step S37 ) . 

It is determined whether processing for all 
keywords is ended (step S31). If processing for all 
keywords is ended, the contents described by the HTML 
are sent to the output or display section (step S38). 
Otherwise, processing is continued. 

FIG. 8 shows an example of a window generated by 
the link section 106 in the above-described manner. In 
this example, keywords in the user profile are 
"shopping" , "public facility" , "transportation/bank" , 
and "health/hospital". Therefore, partial video data 
of programs associated with words such as "department 
store" and "bakery" associated with "shopping" are 
acquired, and representative images each forming one 
frame of partial video data are pasted in line like 
indices. CMs arranged sporadically are advertisements 
of sponsors. CMs can also be selected on the basis of 
matching between program features and the user profile, 
like program information. Hence, CMs best associated 
with the selected multimedia information can be 
selected. In the window shown in FIG. 8, each 
representative image is linked to corresponding partial 



video data such that the partial video data is 
displayed by a click button. 

To generate such a window, a necessary description 
is prepared using HTML. HTML is an abbreviation for 
HyperText Markup Language, which indicates a page 
description language used as the general format of 
information provided by the WWW or W3 (World Wide Web) 
service of the Internet. HTML is based on SGML 
(Standard Generalized Markup Language) and can 
designate the logical structure of a document and link 
between documents by inserting a markup called a "TAG" 
in the document. 

WWW is a client/server information service in the 
Internet. A network user can access information using 
a dedicated Web browser. Provided information are HTML 
documents called homepages, Web pages, or WWW pages 
connected by hyper link. Information can be displayed 
by tracking the link. 

Documents handled by WWW can include multimedia 
information, and the server side can execute a program 
to perform special processing therefore. This function 
can be used to provide a unique information search 
service. 

In the above-described example, HTML documents are 
used to display CMs together with selected programs. A 
method of displaying CMs when displaying selected 
programs or part of programs as video data will be 



described next. 

In the example shown in FIG. 9, a CM banner 
advertisement is displayed together with program 
video data. While displaying video data, a banner 
advertisement is displayed on the lower side of the 
program video data. In this case as well, the window 
can jump to a corresponding WWW page in accordance with 
an instruction from the user. 

As shown in FIG. 10, CM video data may be 
displayed as a subwindow of the program video data 
window. In this method as well, the linked WWW page 
can be used. 

As described above, since a CM best associated 
with a scene of a program or a CM associated with 
user's taste can be selected in accordance with 
keywords associated with a scene of the selected 
program or user's taste, an advertisement can be 
effectively displayed. 

As described above, according to this embodiment, 
at least one of moving image analysis, acoustic/speech 
analysis, and text analysis is applied to the database 
storing multimedia information and multimedia 
information provided from the database, the multimedia 
information are sorted on the basis of the analysis 
result, and the analysis result is managed in units of 
sorts. The analysis result is searched in accordance 
with the user profile, partial information of 



multimedia information according to user's taste are 
selected, and the selected partial images are 
associated with each other, reconstructed, and provided 
to the user. 

According to this embodiment, an information 
providing method and apparatus and an information 
reception apparatus capable of appropriately selecting 
and providing only portions of user's actual interest 
from a number of multimedia information instead of 
filtering in units of programs are provided. This 
eliminates a disadvantage of the prior art in which 
even a program that the user wants to watch only 
partially need be entirely recorded or watched. 

Since CMs can also be stored in a database like 
program information and selected like program 
information, viewers surely watch CMs that the program 
provider wants the viewer to watch. In addition, since 
CMs are selected in accordance with user's taste or 
interest, the effect of the advertisements can be 
increased. This eliminates a disadvantage of the prior 
art in which when only part of video data is extracted 
for recording or watching, CMs that the program 
provider wants the viewer to watch are omitted. 

Other embodiments of the present invention will be 
described below. The same reference numerals as in the 
first embodiment denote the same parts in the following 
embodiments, and a detailed description thereof will be 



omitted. 

When the broadcasting receiving device of a user 
performs the series of processing operations of the 
first embodiment, i.e., processing of analyzing 
multimedia information stored in a database, managing 
features of program information as the analysis result 
using a database, reading out partial information of 
appropriate multimedia information from the database in 
accordance with the user profile, associating them with 
each other, and reconstructing and providing them, the 
processing amount is too large to result in overload. 
To solve this problem, a server/client system is built 
to perform some processing operations in the server. 

FIG. 11 shows a server /client system according to 
the first modification of the first embodiment. In 
this case, the multimedia information database 101, CM 
database 108, feature extraction section 102, program 
feature database 103, CM feature database 109, search 
engine 105, link section 106, and keyboard 110 are on 
the server side, and the display section 107, user 
profile database 104, and keyboard 111 are on the 
client side. 

FIG. 12 shows a server /client system according to 
the second modification of the first embodiment. In 
this case, the multimedia information database 101, CM 
database 108, feature extraction section 102, program 
feature database 103, CM feature database 109, search 
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engine 105, and keyboard 110 are on the server side, 
and the link section 106, display section 107, user 
profile database 104, and keyboard 111 are on the 
client side. 

5 FIG. 13 shows a server/client system according to 

the third modification of the first embodiment. In 
this case, the multimedia information database 101, CM 
database 108, feature extraction section 102, program 
j;3 feature database 103, CM feature database 109, and 

* ( ; 10 keyboard 110 are on the server side, and the search 

snl engine 105, link section 106, display section 107, user 

i u 

i'3 

profile database 104, and keyboard 111 are on the 

Vn 

client side. 

ft 

j;* When the system of the first embodiment is 

I'll 15 constructed using a server/client system, only a 

*,3 section which stores a user profile and a section which 

sends it to the server and a section which receives a 
search result from the server and a section which 
displays it are on the client side, as shown in FIG. 11. 
20 Alternatively, a link section being based on the basis 

of the search result is also arranged on the client 
side, as shown in FIG. 12, or a searching section is 
also arranged on the client side, as shown in FIG. 13. 
The range of functions provided on the client side 
25 depends on the processing capability of the client. 

When all sections except the databases 101, 103, 108, 
and 109 and feature extraction section 102 are on the 
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client side, as shown in FIG. 13 , the client must 
download the processing result. Hence, in this 
arrangement, the range of functions depends on not only 
the processing capability of the client but also the 
information storage capability and line capability for 
download. However, since processing operations can be 
distributed, this arrangement is effective when the 
line is a CATV, an optical fiber, or intranet. 
Second Embodiment. 

In the present invention, the features of video 
data can be managed not in units of programs but in 
units of frames. This enables rating survey in units 
of frames and solves a problem of conventional rating 
survey in units of programs. Hence, viewing history 
usable for analysis of user's taste or interest can be 
obtained . 

FIG. 14 is a block diagram showing the system 
arrangement of the second embodiment capable of 
recording viewing history in units of frames. This 
system has, in addition to the system of the first 
embodiment shown in FIG. 1, a viewing information 
control section 120, history information recording 
section 12 1, reproduction control section 122, and link 
section 123. The viewing information control section 
120 acquires, from the reproduction control section 122, 
viewing information when a user has watched an 
information program (media) and transmits the 
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information to the link section 123. The link section 
12 3 reads out a program feature corresponding to the 
viewing information from the program feature database 
103, corresponds them with each other, and supplies 
5 them to the history information recording section 121. 

The viewing information contains information of the 
watched multimedia information and information 
representing the scenes watched and the number of 
i;3 watches. 

s! =. 10 The history information wherein the viewing 

>. •: 
' % 

fu information and the program feature are corresponded to 

n 

each other may be recorded in the recording section 121 
and simultaneously uploaded to the program provider 
(server) side. Alternatively, the history information 
15 may be uploaded to an external database (database of 

the manager on the server side) when the history 
information for a predetermined period or a 
predetermined amount of the history information is 
recorded in the history information recording section 
20 121. The viewing information and the video data may be 

corresponded to each other to provide the history 
information representing only watched video data, 
watched time sections, and frequencies of watches. 
Alternatively, using section information of a scene of 
25 video data or index information, statistical 

information of watched scenes and frequencies of 
watches may be acquired to provide the history 
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information. 

In a server/client system, the history information 
recording section 121 is preferably located in the same 
site as that of the viewing information control section 
5 120. However, various changes and modifications can be 

made as will be described below. 

FIG. 15 shows a server /client system according to 
the first modification of the second embodiment. In 
r'3 this case, the multimedia information database 101, CM 

10 database 108, feature extraction section 102, program 

j.y feature database 103, CM feature database 109, search 

It engine 105, link section 106, link section 123, and 

*;• 5 
if, 

• fW keyboard 110 are on the server side, and the display 

si 

!'* section 107, user profile database 104, viewing 

|y 

|'U 15 information control section 120, history information 

lu 

u recording section 121 , reproduction control section 122 , 

and keyboard 111 are on the client side. 

FIG. 16 shows a server/client system according to 
the second modification of the second embodiment. In 

20 this case, the multimedia information database 101, CM 

database 108, feature extraction section 102, program 
feature database 103, CM feature database 109, search 
engine 105, and keyboard 110 are on the server side, 
and the display section 107, user profile database 104, 

25 viewing information control section 120, history 

information recording section 121, reproduction control 
section 122, link section 106, link section 123, and 
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keyboard 111 are on the client side. 

FIG. 17 shows a server/client system according to 
the third modification of the second embodiment. In 
this case, the multimedia information database 10 1, CM 
database 108, feature extraction section 102, program 
feature database 103, CM feature database 109, and 
keyboard 110 are on the server side, and the display 
section 107, user profile database 104, viewing 
information control section 120, history information 
recording section 121, reproduction control section 122 
link section 106, link section 123, search engine 105, 
and keyboard 111 are on the client side. 

FIG. 18 shows the processing flow of the link 
section 123. When the viewing information of a viewer 
is supplied from the reproduction control section 122, 
the viewing information control section 120 holds the 
start time and end time of watch. Even when the 
channel is changed, the viewing information control 
section 120 determines an end and start of watch are 
generated. When watch is ended, the end is detected 
(step S41), and the watch start time and the watch end 
time paired with the start time are acquired (step S42) 
The information may be directly sent to the multimedia 
information database 101 or history information 
recording section 121 and recorded (step S43). In that 
case, the recording data is recorded using the ID of 
the video data, start time, and end time, as shown in 



- 33 - 

FIG. 19A. As another method, scenes of video data, 
which were viewed, are acquired by the link section 106 
(step S44). As shown in FIG. 19B, a video data ID and 
a scene ID are processed to provide information of 
5 frequency of watches, and the information is sent to 

the multimedia information database 101 or history 
information recording section 121 and recorded. The 
totalization method is not limited to this. The 

ij frequency may be calculated for each genre or each 

i.3 

10 keyword independently of video data IDs or program IDs. 

fu The frequency can be calculated by counting " 1" when 

j'S the user has watched a scene once or weighting it in 

' accordance with the length of time of watch. 

With this processing, the audience behavior can be 

j y 15 grasped in units of frames, and user's taste or 



*.3 interest can be surely known. 

Therefore, the information in the user profile 
database 104 may be updated on the basis of the history 
information recorded in the history information 
20 recording section 121. FIG. 20 shows this processing 

flow. First, the history information recording section 
121 extracts a scene with a high frequency of watches 
from history information of a user (step S50), and a 
feature (keyword) corresponding to the scene is 
25 extracted (step S51). It is determined whether the 

profile data of the user in the user profile database 
104 has a corresponding keyword (item) (step S52). If 
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YES in step S52, the priority of the keyword in the 
user profile data is raised (step S53). If NO in 
step S52, the item is added to the user profile data 
( step S54 ) . 

5 As described above, according to the second 

embodiment, a program and scenes thereof, which are 
watched by the user, and the number of watches are 
recorded as history information simultaneously as the 
Q viewer watches the program. Since the user profile is 

10 rewritten in accordance with the history information, a 

py user profile that appropriately reflects user's taste 

and interest can be obtained, and information of user's 
interest can be selectively provided to the user. The 
history information can be acquired not in units of 
[M 15 programs but in units of scenes of a program. 

.2 Therefore, the relationship between user's taste and 

the scenes and contents of the program can be analyzed 
in detail. When the history information is auto- 
matically uploaded from the client side to the server 
2 0 side, cumbersome acquisition can be automatically 

performed . 

Additional advantages and modifications will 
readily occur to those skilled in the art. Therefore, 
the present invention in its broader aspects is not 
25 limited to the specific details, representative devices, 

and illustrated examples shown and described herein. 
Accordingly, various modifications may be made without 
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departing from the spirit or scope of the general 
inventive concept as defined by the appended claims and 
their equivalents. 

As has been described above , according to the 
present invention, only video data of portions which 
are actually required by the user who is watching the 
program can be recorded or reproduced without recording 
or reproducing the entire program. In addition, 
partial video data (video data in units of sorts) are 
associated with each other and reconstructed to result 
in visually convenient display. Furthermore, 
commercial messages are also selectively provided in 
accordance with user's taste. Hence, even when only 
part of video data is selected and recorded or watched, 
commercial messages that the program provider wants the 
viewer to watch are not omitted, unlike the prior art. 



